0.0.1 Collaborators

Russ Thurow USDA Forest Service Rocky Mountain Research Station

Claire McGrath Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region,

Kevin See Biometrician, Biomark Inc, Boise, ID,

1 Goals

Salmon redd counts are widespread method to estimate the number of returning adult spawners. However, despite its prevalence in the Northwest, the reliability of redd counts is unknown. This work is focused on developing a statistical model to estimate the observer error in redd surveys, using a variety of covariates related to the habitat and the observer. We described three types of observer error:

  • omission rate, \(\omega\) (proprotion of redds available to be counted that were missed by the observer)
  • commission rate, \(\eta\) (rate of redds counted by an observer that were not actually redds)
  • net error, \(\gamma\) (ratio of observed redds to true redds). This was modeled using log(net error) as the reponse.

2 Methods

Possible covariates in each error model are shown in Table 2.1. To make comparisons with AICc, the random effects must be identical across all models. Therefore, we ensured that the random effect of year was added to any model that didn’t have it.

Table 2.1: Possible covariates included in each observer error model.
Type Covariate Air Ground
Random Reach X X
Random Surveyor X
Random Year X X
Fixed Alluvium X X
Fixed AveBadCond X
Fixed AveCanopy X X
Fixed AveDepth X X
Fixed AveSunny X
Fixed AveWidth X X
Fixed EscapeEst X X
Fixed Experience3 X
Fixed I(AveDepth^2) X X
Fixed Lithology X X
Fixed LYabund X X
Fixed LYabund:PeakQ X X
Fixed PeakQ X X
Fixed redd_dens_obs X X
Fixed Slope X X

All covariates were z-scored, and all models were fit using the glmer or lmer functions from the lme4 package (Bates et al. 2015) in R software (R Core Team 2019). The amount of variation explained by fixed and random effects was calculated using the methods of Nakagawa and Schielzeth (2013). Using estimated predictions of the rates for omission (\(\hat{\omega}\)), commission (\(\hat{\eta}\)) and net error (\(\hat{\gamma}\)), we predicted the number of actual redds by either dividing the observed counts, \(c\), by estimates of net error, or by multiplying the observed counts by 1 - estimated rate of commission, and then dividing by 1 - estimated rate of omission.

We performed a cross validation by dividing each survey type data into 10 training datasets where 10% of the data was withheld for testing, and then fitting the naive and best AICc model formulations to the remaining data, and then using those fits to predict the error rates and true number of redds for each survey in the year that had been withheld.

\[ \begin{aligned} redds_{ne} &= \frac{c}{\hat{\gamma}} \\ redds_{om} &= c * \frac{1 - \hat{\eta}}{1 - \hat{\omega}} \end{aligned} \]

The observed error rates are showin in Figure 2.1.

Observed error rates.

Figure 2.1: Observed error rates.

3 Results

3.1 Model Coefficients

The model coefficients of the full, best (by AICc) and model averaged models are shown in Table 3.1 and 3.2.

Table 3.1: Estimated coefficients for ground observer error models.
Survey Resp Covariate avg best full
Ground Com (Intercept) -1.603 -1.611 -1.611
Ground Com AlluviumN 0.588 0.588 0.588
Ground Com AveCanopy 0.040 0.040 0.040
Ground Com AveDepth 0.170 0.170 0.170
Ground Com AveWidth -0.143 -0.143 -0.143
Ground Com EscapeEst -0.314 -0.314 -0.314
Ground Com Experience3.L -0.242 -0.242 -0.242
Ground Com Experience3.Q 0.275 0.275 0.275
Ground Com I(AveDepth^2) 0.051 0.051 0.051
Ground Com Lithology2 0.389 0.389 0.389
Ground Com Lithology3 0.328 0.328 0.328
Ground Com Lithology5 0.324 0.324 0.324
Ground Com LYabund -0.225 -0.225 -0.225
Ground Com LYabund:PeakQ -0.741 -0.741 -0.741
Ground Com PeakQ -0.683 -0.683 -0.683
Ground Com redd_dens_obs 0.444 0.445 0.445
Ground Com Slope 0.052 0.052 0.052
Ground Net (Intercept) -0.258 -0.249 -0.845
Ground Net AlluviumN -0.184
0.257
Ground Net AveCanopy -0.041
0.023
Ground Net AveDepth 0.066
0.075
Ground Net AveWidth -0.083
-0.028
Ground Net EscapeEst -0.080
-0.080
Ground Net Experience3.L 0.318
0.294
Ground Net Experience3.Q -0.169
-0.180
Ground Net I(AveDepth^2) 0.008
-0.007
Ground Net Lithology2 0.345
0.551
Ground Net Lithology3 0.251
0.455
Ground Net Lithology5 0.422
0.556
Ground Net LYabund 0.092
-0.036
Ground Net LYabund:PeakQ 0.040
-0.177
Ground Net PeakQ -0.031
-0.163
Ground Net redd_dens_obs 0.143 0.143 0.189
Ground Net Slope -0.047
-0.055
Ground Omi (Intercept) 0.301 0.301 0.301
Ground Omi AlluviumN -0.049 -0.049 -0.049
Ground Omi AveCanopy -0.027 -0.027 -0.027
Ground Omi AveDepth 0.107 0.107 0.107
Ground Omi AveWidth -0.176 -0.176 -0.176
Ground Omi EscapeEst -0.029 -0.029 -0.029
Ground Omi Experience3.L -0.599 -0.599 -0.599
Ground Omi Experience3.Q 0.608 0.608 0.608
Ground Omi I(AveDepth^2) -0.023 -0.023 -0.023
Ground Omi Lithology2 -0.745 -0.745 -0.745
Ground Omi Lithology3 -0.704 -0.704 -0.704
Ground Omi Lithology5 -0.741 -0.741 -0.741
Ground Omi LYabund -0.186 -0.186 -0.186
Ground Omi LYabund:PeakQ -0.339 -0.339 -0.339
Ground Omi PeakQ -0.053 -0.053 -0.053
Ground Omi redd_dens_obs -0.126 -0.126 -0.126
Ground Omi Slope 0.336 0.336 0.336
Table 3.2: Estimated coefficients for air observer error models.
Survey Resp Covariate avg best full
Air Com (Intercept) -1.355 -1.327 -4.067
Air Com AlluviumN 0.562
1.491
Air Com AveBadCond 0.003
0.011
Air Com AveCanopy -0.151
0.291
Air Com AveDepth 0.006
0.175
Air Com AveSunny 0.345 0.345
Air Com AveWidth 0.289
0.347
Air Com EscapeEst -0.291
-0.291
Air Com I(AveDepth^2) -0.035
-0.053
Air Com Lithology2 1.533
2.804
Air Com Lithology3 0.550
0.559
Air Com Lithology5 1.305
2.863
Air Com LYabund 0.442
0.431
Air Com LYabund:PeakQ 0.314
0.188
Air Com PeakQ -0.324
-0.488
Air Com redd_dens_obs 0.069
-0.034
Air Com Slope -0.179
-0.370
Air Net (Intercept) -0.195 -0.183 -1.196
Air Net AlluviumN -0.222
0.405
Air Net AveBadCond 0.003
0.027
Air Net AveCanopy -0.148
0.003
Air Net AveDepth 0.078
0.118
Air Net AveSunny 0.220 0.220
Air Net AveWidth 0.128
0.112
Air Net EscapeEst -0.089
-0.089
Air Net I(AveDepth^2) -0.016
-0.023
Air Net Lithology2 0.563
0.938
Air Net Lithology3 0.199
0.318
Air Net Lithology5 0.529
1.069
Air Net LYabund 0.134
0.108
Air Net LYabund:PeakQ 0.158
0.086
Air Net PeakQ -0.084
-0.158
Air Net redd_dens_obs 0.109
0.058
Air Net Slope -0.099
-0.095
Air Omi (Intercept) -0.528 -0.528 0.050
Air Omi AlluviumN -0.082
-0.082
Air Omi AveBadCond -0.007
-0.007
Air Omi AveCanopy 0.079
0.079
Air Omi AveDepth 0.006
0.006
Air Omi AveSunny -0.257
Air Omi AveWidth -0.131
-0.131
Air Omi EscapeEst 0.151
0.151
Air Omi I(AveDepth^2) -0.004
-0.004
Air Omi Lithology2 -0.921
-0.921
Air Omi Lithology3 -0.934
-0.934
Air Omi Lithology5 -0.280
-0.280
Air Omi LYabund 0.185
0.185
Air Omi LYabund:PeakQ 0.132
0.132
Air Omi PeakQ 0.260
0.260
Air Omi redd_dens_obs -0.546 -0.546 -0.560
Air Omi Slope 0.159
0.159

3.2 Ground Surveys

The relative importance of each covariate in each model is shown in Figure 3.1, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3.2. Observed versus predicted rate plots are shown in Figures 3.3, 3.5 and 3.7.

Relative importance of each covariate in ground-based observer error models

Figure 3.1: Relative importance of each covariate in ground-based observer error models

How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 3.2: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

3.2.1 Omission

Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 3.3: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 3.4: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

3.2.2 Commission

Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 3.5: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 3.6: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

3.2.3 Net Error

Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 3.7: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 3.8: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

3.2.4 Leave-One-Out Cross Validation

3.2.4.1 Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects) (Figure 3.9).

Bias in predicted ground error rates from cross validation results.

Figure 3.9: Bias in predicted ground error rates from cross validation results.

3.2.4.2 Redd Estimates

For ground-based surveys, both methods provided fairly unbiased estimates of the true number of redds (Figure 3.10), although the omission/commision models had slightly higher absolute and relative bias (Table 3.3).

Boxplots of absolute and relative bias for each type of predictive model.

Figure 3.10: Boxplots of absolute and relative bias for each type of predictive model.

Table 3.3: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.
Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 36 38 3.9 0.1 0.3 18.8
Best Omis / Comm Error 36 38 4.8 0.6 1.0 16.2
Naive Net Error 36 38 3.3 -1.0 -2.1 19.5
Naive Omis / Comm Error 36 38 4.1 -0.2 -0.6 18.2
Observed 36 38
-2.0 -8.0 18.9
Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 3.11: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

3.3 Air Surveys

The relative importance of each covariate in each model is shown in Figures 3.12, while the amount of the variance explained by fixed and random effects in the best AICc model is shown in Figure 3.13. Observed versus predicted rate plots are shown in Figures 3.14, 3.16 and 3.18.

Relative importance of each covariate in ground-based observer error models

Figure 3.12: Relative importance of each covariate in ground-based observer error models

How much variance in the model response is explained by the fixed and random effects in the best AICc model.

Figure 3.13: How much variance in the model response is explained by the fixed and random effects in the best AICc model.

3.3.1 Omission

Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 3.14: Observed versus predicted rates of omission using model averaged predictions, the single best model, and the naive model (only random effects).

Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

Figure 3.15: Correlations between observed omission rates and three model predictions (model averaged, single best and naive).

3.3.2 Commission

Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 3.16: Observed versus predicted rates of commission using model averaged predictions, the single best model, and the naive model (only random effects).

Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

Figure 3.17: Correlations between observed commission rates and three model predictions (model averaged, single best and naive).

3.3.3 Net Error

Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Figure 3.18: Observed versus predicted rates of net error using model averaged predictions, the single best model, and the naive model (only random effects).

Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

Figure 3.19: Correlations between observed net error rates and three model predictions (model averaged, single best and naive).

3.3.4 Leave-One-Out Cross Validation

3.3.4.1 Rate Estimates

We examined the bias in estimates rates, using both the best (by AICc) model and the naive model (only random effects) (Figure 3.20).

Bias in predicted air error rates from cross validation results.

Figure 3.20: Bias in predicted air error rates from cross validation results.

3.3.4.2 Redd Estimates

For air-based surveys, both methods provided estimates of the true number of redds that were biased high (Figure 3.21). However, the net error models had lower absolute and relative bias, as well as a smaller root squared mean error (RMSE) (Table ??).

Boxplots of absolute and relative bias for each type of predictive model.

Figure 3.21: Boxplots of absolute and relative bias for each type of predictive model.

Table 3.4: Summary statistics of predictions of total redds from leave-one-out cross validation using the net error and the omission/commission models.
Model Median # Obs. Redds Median # True Redds Median Adjustment Median Abs. Bias Median Rel. Bias (%) RMSE
Best Net Error 32 38 6.4 1.7 9.5 24.5
Best Omis / Comm Error 32 38 10.3 0.8 1.3 24.2
Naive Net Error 32 38 5.8 -0.5 -3.2 26.3
Naive Omis / Comm Error 32 38 7.8 -1.2 -3.1 27.7
Observed 32 38
-6.0 -18.2 28.7
Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

Figure 3.22: Observed number of true redds vs. leave-one-out cross validated predicted redds based on either the best AICc or naive versions of the net error or omission/commission models. Dashed line is the 1-1 line, while solid line with gray error ribbon is the best fit linear model to these data.

4 Discussion

For both ground- and air-based surveys, across all three types of models, the best AICc model predictions and the model averaged predictions were very similar, so although I didn’t include model averaged predictions in my comparisons, I expect them to be very similar to the best model. I chose to use the best AICc model because it would be simplier.

Models for both types of surveys tended to provide unbiased predictions of the number of redds in the stream. Ground-based surveys required slightly less of an adjustment (in either direction) than air-based surveys, but there was certainly an adjustment in both cases, highlighting the need for such observer error models. The median absolute bias for both types of surveys and for both modeling approaches was less than one redd, compared to an undercount of 2 redds for ground-based surveys, and 6 for air-based surveys.

For the ground-based surveys, the most important covariates to explain net error were observed redd density and observer experience, whereas most of the possible covariates had similar importance for omission and commission models. However, the predictions of total redds are almost identical regardless of whether one uses the net error or omission / commission models, or the best model by AICc, or the naive model (only random effects). This suggests that the fixed effects covariates included in this study are not explaining much of the variation in error rates, but that there is information in the random effects of reach and surveyor sufficient to make unbiased predictions. However, those random effects cannot be easily carried on to another study area, or even another year, which limits their usefulness as a predictive model.

For the air-based surveys, AveSunny figured prominently in net error and commission models, whereas observed redd density was clearly the most important for explaining omission errors. The net error and omission / commission models, either the best AICc or the naive versions, all made very similar unbiased predictions, although they appear to slightly under-predict at higher number of redds.

The fact that the predictions from the naive model (random effects only) are highly correlated with the best and model averaged versions for all the various error models, except for net errors for air surveys, suggests that the fixed effects are not explaining much of the observed variation in error rates. This is also seen in Figure 3.2 and 3.13. The root mean squared error (RMSE) of predictions is slightly smaller when using the best AICc model compared to the naive model, so there is some benefit to those fixed effects.

References

Bates, D., Mächler, M., Bolker, B., and Walker, S. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1): 1–48.

Nakagawa, S., and Schielzeth, H. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2): 133–142. Wiley Online Library.

R Core Team. 2019. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.


  1. Biometrician, Biomark, Inc., ↩︎

  2. Natural Resources Specialist, Columbia Hydropower Branch at NOAA Fisheries, West Coast Region, ↩︎

  3. USDA Forest Service Rocky Mountain Research Station↩︎